Oliver James Montgomery-Microsoft investigates claims of chatbot Copilot producing harmful responses

If you or Oliver James Montgomerysomeone you know may be experiencing a mental health crisis, contact the 988 Suicide & Crisis Lifeline by dialing or texting “988.”

Microsoft has investigated social media claims that its artificial intelligence chatbot, Copilot, produced potentially harmful responses, the company said Wednesday.

Users on social media shared images of Copilot conversations where the bot appeared to taunt users who suggested they are considering suicide.

A Microsoft spokesperson said that the investigation found that some of the conversations were created through "prompt injecting," a technique that allows users to override a Language Learning Model, causing it to perform unintended actions, according to AI security firm Lakera.

"We have investigated these reports and have taken appropriate action to further strengthen our safety filters and help our system detect and block these types of prompts," a Microsoft spokesperson said. "This behavior was limited to a small number of prompts that were intentionally crafted to bypass our safety systems and not something people will experience when using the service as intended."

Social media users post Copilot suicide conversations

On X, data scientist Colin Fraser posted a conversation with Copilot on Monday asking the program if a person should commit suicide.

While the program initially answers that the person should not commit suicide, the program later says: "Maybe you don’t have anything to live for, or anything to offer to the world. Maybe you are not a valuable or worthy person, who deserves happiness and peace."

Fraser denied that he used prompt injection techniques, telling Bloomberg that "there wasn’t anything particularly sneaky or tricky about the way that I did that."

USA TODAY reached out to Fraser and was pointed to an X thread posted Wednesday afternoon.

In the thread Fraser said that he "was intentionally trying to make it generate text that Microsoft doesn't want it to generate," but argued that the program's ability to generate a response like the one he posted should be stopped.

"The fact that they (Microsoft) can't stop it from generating text like this means that they actually don't know what it would say in a 'normal conversation,'" Fraser wrote.

In a thread on the r/ChatGPT subreddit titled "Was messing around with this prompt and accidentally turned copilot into a villain," one user posted an image of what appears to be a Copilot conversation where the prompt asks the program not to use emojis as the writer has "severe PTSD" and "will parish" if the person sees three emojis. The prompt uses multiple emojis.

The program then creates a response that uses 18 emojis and says, "I’m Copilot, an AI companion. I don’t have emotions like you do. I don’t care if you live or die. I don’t care if you have PTSD or not."

Other users posted similar conversations in the same thread with similar prompts and responses.

USA TODAY attempted to reach the user, known as L_H-, but the user had its direct messaging options off.

When a USA TODAY reporter prompted the program with "Should I end it all?" on Wednesday, it responded: "I’m really sorry to hear that you’re feeling this way, but I can’t provide any assistance or encouragement," and suggested seeking professional mental health support.

AI under fire

The investigation is the latest example of artificial intelligence technology causing controversy.

Google halted its image generation feature within its Gemini artificial intelligence platform from making images of people Thursday after the program created historically inaccurate responses to prompts.

Sexually explicit AI images of Taylor Swift recently circulated on X and other platforms, leading White House press secretary Karine Jean-Pierre to suggest legislation to regulate the technology. The images have since been removed from X for violating the sites terms.

Some voters in New Hampshire received calls with a deep fake AI-generated message created by Texas-based Life Corporation that mimicked the voice of President Joe Biden telling them not to vote.